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Effective Use of Graduating Senior Survey as Part of Program Assessment 

Abstract 

This paper presents a case study, including a survey sample, of the process used to 
implement an effective Graduating Seniors Survey within the institutional effectiveness (IE) 
process at a major metropolitan research university. Organizationally, the university survey 
support office and the faculty and staff responsible for the assessment processes jointly 
implement an integrated IE process for continuous improvement. Methodologically, the 
Graduating Seniors Survey directly supports the assessment process by collecting student 
feedback from graduating seniors at the program level. The survey support office thereby 
provides each program with the specific data necessary to measure its assessment 
objectives. 
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Introduction 

Continuous quality improvement has become a significant component of the culture 
of American higher education (Astin, 1991; Ewell, 1984; Seymour, 1993; Sims & Sims, 

1995). Tierney (1999) and others have maintained that colleges and universities must 
become responsive to internal and external sources that call for organizational change. 
Seymour (1993) cited four driving forces behind the concern for quality improvement in 
higher education administration and services. These pressure points include increased 
competition for tuition dollars and state subsidies, rising costs, increased state reporting to 
ensure accountability, and a service orientation that regards students and other constituents 
(i.e., alumni, parents and employers) as customers. Recently, accreditation agencies (e.g., 
Southern Association of Colleges and Schools, 1997) have strongly encouraged higher 
education to implement continuous quality improvement that features institution-wide 
planning and assessment processes to assure information-driven decisions at all levels of 
the organization. 

A process to support quality improvement was developed over fifty years ago by 
W.A. Shewhart at Bell Telephone Laboratory (Senge, 1991). His Plan-Do-Check-Act cycle 
emphasized a process-oriented way of thinking that does not have an end point, but is 
circular and creates a cycle of continuous improvement. Creating an infrastructure to 
support a quality improvement process in higher education is essential to the success of the 
endeavor. Several necessary infrastructure elements have been identified (Pet-Armacost & 
Armacost, 2002; Seymour, 1994; Simms & Sims, 1995): (a) cultural change, (b) commitment 
from upper administration, (c) implementation of continuous planning and assessment that 
uses existing organizational structure, (d) participation at all levels of the organization, (e) 
resources and staff to support the process, and (e) a long term commitment. Sims & Sims 
(1995) assert, “In the initial stage, an effort must be mounted to begin to change the culture 
of the institution. Unless a culture is based on customer satisfaction and continuous 
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improvement and teamwork is established, Total Quality Management will be of little more 
than just another one of administration’s programs.” (p.1 2) 

Equally important is the commitment from upper administrators. Sims & Sims (1995) 
advises top administrators to embrace quality improvement as sound strategy that can 
improve the effectiveness of the university and to create an infrastructure to support the 
process. They stated: 

...ensure that quality is managed as the institution’s other most important tasks are 
managed and not assume that quality will manage itself; establish specific management 
direction and goals for results, then expect, monitor, and reward progress; use the existing 
institutional structure and involve everyone (that is, administration, faculty, staff, and 
students); and use systematic institutional change process with an institutional-specific self- 
designed plan... Quality efforts have a greater likelihood of success in institutions of higher 
education when a supporting infrastructure is included in the quality improvement process 
itself. Who will be responsible for developing a quality mission, a quality education program, 
and a plan for institutional change? How and when will it be staffed? What information and 
data requirements exist? (p. 14) 

A critical component of an institutional effectiveness assessment plan 
implementation is the capacity to measure objectives at the institution, program, and unit 
level. (Astin, 1991, Ewell, 1984; Seymour, 1993; Seymour, 1994). The institution needs 
access to valid, reliable and accessible sources of information that measure how students, 
alumni, employers, faculty and staff perceive the quality and effectiveness of programs and 
services. This measurement effort needs to be ongoing and continuous, rather than one- 
shot studies. It also needs to be integrated with other university information sources such 
as the student and employee databases, measures of learning competencies, and 
community and employer needs assessments. Collection and dissemination of reliable and 
ongoing student assessment information enables academic programs and service units to 
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measure annual objectives, use the results to monitor the objectives, make changes as a 
result of this information and use ongoing assessment information to monitor the 
effectiveness of any changes made (Banta, 1997; Ewell, 1984; Tierney, 1999). 

In 1997, The University of Central Florida (UCF) initiated a needs assessment of the 
institutional information and desired capabilities related to institutional research and planning 
(Pet-Armacost, Armacost & Young, 2001a). The UCF 21 Operational Excellence Initiative 
lead to a renewed commitment to quality enhancement and continuous improvement, and 
processes related to planning and assessment. Restructuring of the organizational 
infrastructure that greatly expanded support of UCF’s continuous improvement process was 
launched in March 2000 (Pet-Armacost, Armacost & Young 2001b). The newly established 
Office of Operational Excellence and Assessment Support (OEAS) provides support to 
monitor the institutional effectiveness process and assists academic and administrative units 
in the development and implementation of assessment plans. It also provides ongoing 
survey and statistical support, special studies and process analysis support. 

The survey support and statistical studies staff of OEAS provide collaborative 
service-oriented support to academic programs, administrative units, and upper 
administration to satisfy various information needs. A primary initiative is to create processes 
and provide support to assure continuous, reliable, valid and accessible measures of how 
students, alumni, employers, faculty and staff perceive the quality and effectiveness of 
programs and services. These information sources are needed for quality improvement and 
quality assurance efforts such as: (a) the annual institutional effectiveness planning and 
assessment cycle, (b) program review, (c) accreditation self-studies, (d) state 
accountability measures and (e) institution level strategic objectives. 

Prior to March 2000, UCF conducted periodic surveys of students, alumni, and 
employers and many programs and units conducted their own surveys to obtain information. 
These efforts, while admirable, often fell short of satisfying needs. The university surveys 
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did not provide data at the program level, response rates were poor, and individual 
programs lacked the infrastructure to design valid surveys, obtain a representative sample, 
and compile and analyze data into useable information. 

This paper describes the implementation of an ongoing university-wide Graduating 
Seniors Survey as a component of a comprehensive plan to provide accessible information 
sources to the UCF community to support assessment processes. To optimize buy-in, 
ongoing collection, dissemination and use of the information, the survey design and 
collection processes were closely integrated with UCF’s institutional effectiveness (IE) 
planning and assessment process (for a complete description of the UCF IE assessment 
process see Albert & Pet-Armacost, 2002). The next section describes the organizational 
structure that makes the process work. Then the design, administration and use of results 
from the Graduating Seniors Survey and future developments are described. 

Process Integration and Organization: Key to Continuous Measurement and Use 

The organizational relationship between the university survey and statistical staff and 
the faculty and staff engaged in assessment is paramount to the success of establishing an 
ongoing Graduating Seniors Survey in support of the Institutional Effectiveness (IE) process. 
In this case, a University Assessment Committee (UAC) supports the IE process in an 
advisory capacity. The UAC's role is to provide university-wide leadership for institutional 
effectiveness. With support from OEAS, the committee designs the IE assessment process 
and conducts annual reviews of program and unit plans and assessment results to provide 
feedback for improvement. The UAC works on a continuous basis with College and 
Administrative Divisional Review Committees that work closely with faculty and staff 
engaged in the IE assessment process. A major part of the mission of the university survey 
and statistical support staff is to improve the quality of university operations and academic 
programs. This mission is carried out through direct support of the UAC’s IE assessment 
process. The collaborative work of these two entities made possible an institutional shift 
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from a periodic survey to a ongoing Graduating Seniors Survey that allows data to be 
aggregated to address institution-wide objectives related to instruction and services and 
disaggregated to provide program level information. 

The Transformation and Use of the Graduating Seniors Survey 

There were three main phases to the development and use of a Graduating Seniors 
Survey within the Institutional Effectiveness (IE) process: 

1. Survey design 

2. Survey administration and 

3. Analysis, dissemination and use of the results. 

Each of these phases is briefly described below. 

Survey Design 

In order to integrate the Graduating Senior Survey within the IE process, a survey 
had to be designed that met the needs of the entire university community at all levels of the 
institution. UCF had an existing survey that was in use for the previous five years that 
formed the starting point for the revisions process. The university survey and statistical 
support staff made some changes to the survey unilaterally. These revisions focused on the 
format of the likert rating scales, the question ordering, and removal of some questions after 
a factor analysis revealed high item correlations. This new survey draft was then presented 
to members of the University Assessment Committee (UAC) for their review. The UAC 
members also sought input from faculty within their colleges and staff within administrative 
units. The input from the UAC focused on question content. Revisions to the content were 
made to ensure that the questions accurately represented the assessment objectives of the 
university. Figure 1 a-d shows the UCF 2001-2002 Graduating Seniors Survey instrument. 
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Term Graduating 
CD Summer 2001 
a Fall 2001 
E3 Spring 2002 
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2001- 2002 Questionnaire 
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Figure la. 2001-2002 UCF Graduating Seniors Survey Instrument 
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Figure 1b. 2001-2002 UCF Graduating Seniors Survey Instrument 
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Are you aware oi the services offered by the UCF Ombuds Office? 
r=j Yas 
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Please choose up to 3 sources from which you received 
the MOST BENEFICIAL academic advising during your 
last 2 years at UCF. [SELECT UP TO 3] 

Advisors in my major/coltega 
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CD Student advisors 
f ;d Friends 
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LD Printed material (including the catalog) 

CD | did not saak help from advisors 
CD Other [SPECIFY) 



11 . 



If you received help from your Advisor in your Major/College 
(selection one above), please rate the following aspects of 
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CD Pert-time (less than 12 credit hours per semester! 

14. At which UCF campus did you take most of your course work? 
UCF Breverd Aree Campus 
UCF Daytona Boach Campus 
UCF Orlando (Main) Campus 

Other location [SPECIFY] ^ 

If about equal at each campus, please indicate which campuses. 



CD 

CD 

CD 



2.0 - 2.4 
2.5 * 2.9 
3.0 -3.4 
3.5 -4.0 
Don’t know 



20 . 



If you Intend to engage in further formal study, what is the 
highest degree you eventually expect to obtain? 



CD 

LD 

CD 

CD 

CD 



No further study intended 
Master's degree 

Specialist degreo (J.D., Ed.S., etc.) 
Medical dogree (M.D., D.D.S., etc.) 
Doctorate (Ph.D., Ed.D., etc.) 

Other [SPECIFYI 
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15. 



About how long was your usual commute to the primary 
campus indicated above? 

Cl Usually 0 minutes (lived on campus) 

CD Usually 10 minutes or loss 
CD Usually 1 1 to 20 minutos 
CD Usually 21 to 30 minutes 
C2i Usually more than 30 minutes 



If you are NOT finishing your degree in 4 years, please ■ 

indicate all the reason(s) why not. [SELECT ALL THAT ■ 

APPLY] ■ 

CD I'm in a 5 year degree program 1 

CD I had to withdraw during e semester(s) I ■ 

CD I took e semester(s) off from school ! 1 

CD I took a reduced course load to concentrate on academic grades ■ 

CD My job caused me to take reduced course loads 1 

a I voluntarily took fewer courses to have more time for activities ■ 

CD I changed majors 1 

CD I experienced academic problems 1 

CD I experienced financial problems 1 

CD I experienced personal or family problems j ■ 

CD I was misinformed by advisor(s) j 1 

CD I failed to seek advisor's help i 1 

CD My required courses were not available • 1 

CD Other [SPECIFY] !| 



Figure 1c. 2001-2002 UCF Graduating Seniors Survey Instrument 
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Which, if any, of the following extra-curricular activities 
did you participate in while working on your degree? 
[SELECT ALL THAT APPLY] 

CD Student government 
CD Intercollegiate athletics 
CD Student publications 
f.D Sororities or fraternities 
CD Political activities 
CD Community service 
VIZ Spiritual/roligious activities 
O Performing arts 
CD intramural sports 
CD Recreation and wellness 
CD Organizations related to major 
CD Other UCF clubs and organizations 
CD Other [SPECIFY] 



; 23. Do you feel you could ask a faculty or staff member for 
each of the following types of assistance? 



: 23.1. A letter of recommendation CD Yes CD No 

.23.2. Advice about career decisions CD Yes CD No 

23.3. Advice about personal decisions CD Yes CD No 



I 24. To what extent would vou say vou developed close 
| personal friendships at UCF? ! 

CD Almost all of my closest friends are from UCF 
CD Most of my closest friends are from UCF j 

| CD About half of my closest friends are from UCF, half from elsewhere 

CD Most of my closest friends are from elsewhere 
CD Almost all of my closest friends are from elsewhere j 

25. Please indicate your sex: 

| CD Male CD Female ! 

| 

26. Please indicate your age category: 

CD Less than 24 CD 40 to 49 j 

; CD 24 to 29 CD 50 or older ; 

! CD 30 to 39 

I 

27. Please indicate those racial/ethnic group(s) that apply to you. | 

! [SELECT ALL THAT APPLY] j 

CD African-American/Black 
CD Asian 

CD Caucasian/White 
CD Hispanic or Latino 
r~ i Native American/Alaskan Native 
j CD Other Pacific Islandor/Native Hawaiian 



j Finally, please share your comments and suggestions as requested. 
28. In what way did UCF best meet your expectations? 



29. What one change would you suggest to most improve UCF for other students? 

i ” 



i 

30. Please add any other comments or suggestions regarding your experiences at UCF. ! 



Program Specific Questions 

It. DD m m GO DO CD CD OD <30 US! 



1. CD C2D L'30 03 CD CD OJ CU CEOtt) 

2 . ltd nn cn aa cn do cn txi cm cm 

3 . cnoucn od cd □□ cn cm ra am 

4. CX.I CD CSS CD CD 03 CD CO LT1 02) 

5. art ca co cn :sd cm an on lm leb 



6 . co cm m r:iD cn cm m ran w: nn 

7 . co cn ljo lyj cn cn cn an do on 
6 . qd cn c a co cn do cn an do m 
9 . t:o cn cn ca us.) m cn cn cn oai 

10. CD CD C2D CD CD LX) CD CD CX UK 
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In addition to the revisions to the main body of the Graduating Seniors Survey, space 
for twenty close-ended and three open-ended questions were included in the survey for 
academic programs to add questions specific to their assessment needs (see Figure Id). 
Prior to the launch semester (Spring 2001), through communication by the UAC members, 
faculty and staff from 18 programs worked with the OEAS survey and statistical support staff 
to add program specific questions to the Graduating Seniors Survey. 

In the 2001-2002 academic year, an additional 13 programs have worked with staff 
to design program specific questions for the survey. Increased visibility of OEAS has 
expanded the referral network. With increased incidence, compared to the first year, faculty 
are making direct contact with survey and statistical support staff, and see the program- 
specific question option as a viable and preferred mechanism of obtaining information from 
their graduates. Table 1 shows the major data elements in use by UCF programs. 

Table 1 

Program Specific Data Elements Used in Graduating Seniors Survey 



Program Data Elements 



Overall educational experience in program 


Quality of faculty academic and career 


Quality of instruction 


advising 


Availability and breath of courses 


Plans after graduation 


Perceived competency learning outcomes 


Work Applied In field/Out of field Offer 


Knowledge 


Graduate or professional school 


Skills 


Applied Program of study Acceptance 


Abilities 


Membership professional organization 


Preparation for career or further 


Strengths 


study 


Recommendations for improvement 
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Survey Administration 

In order for the survey to meet the needs of individual programs, it was essential to 
conduct a census survey every semester and obtain as close to a 100% response rate as 
possible, because some programs only have a few graduates. Accomplishing a census 
methodology required the development of a more effective method for data entry. In the 
past the survey had been coded manually. 

The OEAS survey and statistical support staff designed the instrument as a Scantron 
form in order to facilitate the administration and processing of the new Graduating Seniors 
Survey. Use of the Scantron forms facilitated the processing of the data so that the 
Graduating Seniors Survey is now conducted every semester, as opposed to just in the 
spring semester as was done in the past. This form contained the questions to be answered 
by all students, as well as space to fill-in program specific questions. The program specific 
questions were administered on a separate sheet of paper for the answers to be bubbled-in 
on the Scantron form. 

Survey instruments were printed by Scantron and distributed to the college student 
support offices by the survey and statistical support staff. All students are required to visit 
their college office to pick up their “Intent to Graduate” form. Students were asked to 
complete the surveys at the time they visited the college offices to fill-out their Intent to 
Graduate form. The college offices were responsible for administering the program specific 
questions, where applicable. Once the time period for students to apply for graduation was 
over, the college offices sent the completed surveys back to the survey and statistical staff 
for processing. 

With the aid of UAC members, the relationship between the survey and statistical 
support staff and the college student support offices was established. Survey and statistical 
support staff maintained these college partnerships through periodic meetings and e-mails 
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with student support office staff and college deans. This outreach assured quality of 
implementation and enhanced a sharing of best practices among the colleges, as initial 
survey administration effectiveness varied. 

Analysis, Dissemination and Use of Results 

After the survey results were collected and processed, the data were available to 
support the Institutional Effectiveness process. To be effective, the survey results must be 
provided directly to the programs and administrative areas and they should only receive the 
results relevant to them. In the past, the surveys were only summarized at the aggregate 
and college levels and were not useful to the programs. A standard statistical package 
(SPSS) was used for analysis. Most critical were the results from the program specific 
questions. The programming work for the OEAS survey and statistical support staff was 
somewhat extensive, as the variables from the program specific questions had to be read 
into new variables depending on the program. For example, the answers from those who 
filled-out the program specific questions for Philosophy had to be separated from those who 
filled-out the questions for Computer Science. Once accomplished, the data were 
invaluable to the assessment process. Due to the strong ties between the survey and 
statistical support staff and the UAC, programs were provided with results that exactly fit the 
objectives of their assessment plans. 

Tabular results from the Graduating Seniors Survey were shown in WORD 
documents and delivered electronically as e-mail attachments to UAC members for 
distribution to colleges and programs. The distribution contained university-level aggregated 
findings (totals) and breakouts by college. Program-specific results were also delivered 
electronically to faculty program contacts. Aggregated results by college were distributed to 
college deans. Administrative unit directors received results pertinent to their area with 
results broken out by college. 
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Academic programs and administrative units used the information from the 2001- 
2002 Graduating Seniors Survey for the 2000-2001 IE assessment cycle. The UAC review 
of IE assessment results and planned use of the findings in 2001, prompted other academic 
programs to expand their use of this information source in their 2001-2002 IE assessment 
plans. Academic programs and administrative units use assessment results to implement 
changes, make plans for deeper analysis, or changes in measurement strategy that are 
reported in their IE assessment results. The following website provides additional details: 
www.oeas.ucf.edu/ Assessmentlnstitutionaleffectiveness.html. 

Access to program-specific survey questions has made a huge difference to some 
programs and administrative units. For example, the Civil Engineering 1997 plan only 
included the measurement of three program outcomes. Today it includes 10 objectives with 
multiple measures, many of which are based on surveys that are designed and administered 
by the OEAS office. Their current plan shows a maturation of program objectives that is a 
product of increased infrastructure to support the IE process, and a results driven 
assessment process. Until this past year, the Civil Engineering program did not have 
access to survey information specific to their program. 

Challenges and Future Directions 

Implementing an ongoing Graduating Seniors Survey that provides information at the 
university and program level is labor intensive and requires a university commitment of staff 
and resources. The process described here is manual. Accurate tracking of university 
programs and administrative units that would benefit from university and program-level 
information and proactive dissemination is imperative. This tracking task also involves 
matching the timing of the survey support process with the internal IE assessment cycle and 
external accreditation demands. Equally important is responsive staff committed to changing 
the university culture and with vigilance seizing opportunities to subtlety produce attitude 
change that favorably move the institution into a continuous improvement mode of thinking 



Graduating Seniors Survey 16 



and doing. By example, survey support staff must provide customer-driven assistance at all 
phases of program specific and university level survey design, analysis and dissemination. 
Meeting emerging unmet demand for assistance is a major challenge for staff. A sense of 
humor and steadfast belief in the value of customer-driven continuous improvement are 
invaluable when requests pour in, multi-tasking seems like pure theory and deadlines loom. 
Other challenges include linking data sets as the survey instrument evolves from year to 
year and producing reports that highlight university-level trends that provide information in 
support of strategic objectives. 

Thus far, the survey and statistical support staff has provided mostly summarized 
descriptive results as the manual nature of the survey process is restrictive. We have begun 
planning a web-based dissemination process that will free up staff time and provide decision 
makers with dynamic access to survey information. Features of this data mart include a drill 
down capacity to view data at different units of analysis and perform tests of association. 
Linking survey data sets to other information sources (i.e., student and employee 
databases) is another essential goal. The data mart will produce shifts in staff time on task 
that will allow implementation of other continuous measurement survey initiatives (e.g., 
student engagement and satisfaction, alumni, employer, and faculty & staff climate surveys). 
With this integrated web-based information source, survey and statistical support staff can 
move beyond summarizing data to producing higher order analyses, trend analysis and 
exploratory analysis that will provide useful information to the UCF community in support of 
continuous improvement and change. 
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